Recognizing Material Properties from Images

Humans implicitly rely on properties of the materials that make up ordinary objects to guide our interactions. Grasping smooth materials, for example, requires more care than rough ones, and softness is an ideal property for fabric used in bedding. Even when these properties are not purely visual (softness is a physical property of the material), we may still infer the softness of a fabric by looking at it. We refer to these visually-recognizable material properties as visual material attributes. Recognizing visual material attributes in images can contribute valuable information for general scene understanding and for recognition of materials themselves. Unlike wellknown object and scene attributes, visual material attributes are local properties. “Fuzziness”, for example, does not have a particular shape. We show that given a set of images annotated with known material attributes, we may accurately recognize the attributes from purely local information (small image patches). Obtaining such annotations in a consistent fashion at scale, however, is challenging. We introduce a method that allows us to solve this problem by probing the human visual perception of materials to automatically discover unnamed attributes that serve the same purpose. By asking simple yes/no questions comparing pairs of image patches, we obtain sufficient weak supervision to build a set of attributes (and associated classifiers) that, while being unnamed, serve the same function as the named attributes, such as “fuzzy” or “rough”, with which we describe materials. Doing so allows us to recognize visual material attributes without resorting to exhaustive manual annotation of a fixed set of named attributes. Furthermore, we show that our automatic attribute discovery method may be integrated in the end-to-end learning of a material classification CNN framework to simultaneously recognize materials and discover their visual material attributes. Our experimental results show that visual material attributes, whether named or automatically discovered, provide a useful intermediate representation for known material categories themselves as well as a basis for transfer learning when recognizing previously-unseen categories.